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ABSTRACT 

An evaluation procedure. Cognitive Congruence 
Procedure (CCP) , is described which systematically looks at the 
cognitive dimension of instruction and provides information on: (1) 

Congruence of intent and practice, (2) Emphasis given to each kind of 
thinking operation, (3) The degree to which a program has been 
implemented, (4) Judging equivalence of course selections and 
comparing different courses. Several indices are collected: (1) The 

instructor's objectives and test questions, classified using the 
Taxonomy of Intellectual Abilities, an adaptation of Bloom's 
Taxonomy. This provides one index of intended and actual cognitive 
emphasis of the course. (2) The Cognitive Activities Rating Scale 
(CARS) , also based on Bloom's Taxonomy, lists class activities 
calling for each kind of thinking operation. The teacher responds to 
CARS in terms of "Ideal" class activities. Students respond in terms 
of the most characteristic course activities. Comparison reveals the 
congruence of intent and practice. Studies support the accuracy, 
validity, and reliability of each step in the procedure. A teacher 
can use the CCP in modifying practices, expectations, and his 
perception of the instructional process. It should not, however, be 
used to evaluate teachers since many other teaching dimensions not 
contained in the CCP are needed for this purpose. CCP, first used^ 

classes, has been modified for grades six and above in 
Gifted Program Evaluation- 
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ASSESSING INTENT AND PRACTICE IN INSTRUCTION 



A form of course evaluation students have used for years is the matching 
of an instructor’s talk with his behavior. The teacher is judged by comparing 
his words with his actions: Is he seen as practicing what he preaches, or 

does his behavior violate the concepts he teaches? This judgment is made 
subjectively, seldom disclosed to the teacher, and based on criteria which 
vary from student to student. 

1 

Now an evaluation procedure has been developed which provides an objec- 
tive and systematic way for the teacher — and others — to look at the cogni- 
tive dimension of instruction in terms of intents and practices. The procedure 
also provides a way of comparing instructional groups in terms of cognitive 
emphasis . 

Traditional curriculum evaluation calls for the assessment of student 
outcomes. However, outcome evaluation is often a highly inappropriate activity 
as it is presently conducted. Before attempting to determine the effects of 
an instructional treatment, it is important to determine if a treatment actually 
exists. The equivocal results of much instructional research are probably due 
in large part to the assumption that a treatment was fully and adequately per- 
formed. The procedure described herein provides an i’"dex of what the intents 
and practices of a program are and reveals the degree to which they are consistent 

■ ■ ■' 111 -.- - 

^Steele, Joe M. Things As They Are; An Evaluation Procedure to Assess 
Intent and Practice in Instruction . Unpublished doctoral dissertation, Urbana: 
University of Illinois, 1969. 
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and congruent. When it has been determined that a consistent set of intents 
and a matching set of practices are being applied in a course of instruction, 
then it is both necessary and worthwhile to engage in outcome evaluation. 

This evaluation procedure (known as the CC or Cognitive Congruence Pro- 
cedure) is based primarily on the theoretical frameworks for evaluation developed 
2 3 

by Stake and Scriven. Their contributions have led to a major expansion and 
redefinition of the field. There are many dimensions of the classroom which 
are now seen as important to describe and consider in the total evaluation of 
a program. 



FIGURE 1 

Stake's Framework to Guide the Collection of Evaluation Information 



RATIONALE 




n 

Stake, Robert E. The Countenance of Educational Evaluation, Teachers College 
Record , 68, 7, April 1967. 

^Scriven, Michael. The Methodology of Evaluation, Perspectives of Curriculum 
Evaluation, American Educational Research Association Monograph Series on Curricu- 
lum Evaluation. Chicago: Rand McNally and Company, 1967. 
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Figure 1 shows Stake’s model of evaluation, indicating the data to be 
collected in a total evaluation. This guide fociises attention on the context, 
the conditions surrounding the use of an instructional treatment and the trans- 
actional nature of the teaching-learning process. This model itself is not 
static. It is highly dependent on both time and perspective. The process of 
description and the comparison of intents and observations result in 
revisions and new descriptions of the program under evaluation. Those under 
evaluation will react to their own description " - clarifying intents, reassessing 
the situation, and striving for greater consistency and congruence. 

Within Stake’s framework the CC Procedure represents only a part of the 
total evaluation. Transactions intended and observed in the Description Matrix 
are the major concern of this procedure. Furthermore, within this area the 
procedure deals primarily with only one dimension; the cognitive domain. The 
CC Procedure is intended to be applied by teachers and administrators for the 
improvement of the instructional program in use. The intended focus of the 
procedure is on clarifying the program’s objectives and strengthening the rela- 
tionship of instructional practices to those objectives. 

The CC Procedure was first developed and illustrated using six sections 
of a university undergraduate course. Reliability and validity studies were 
undertaken to support the classification system and rating instrument used 
in the procedure. Further studies have since been conducted to allow the 
procedure to be used with public school classes of grade six and above. This 
modification of the procedure is currently being used in the Illinois Gifted 
Program Evaluation. Noncognitive dimensions assessing classroom climate, stu- 
dent attitudes and behavior have been added. 
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Indices of Intent and Practice 



One index of the teacher’s intents in a course of instruction is simply what 

he says he hopes to achieve. These statements are cslled his goals, objectives, 

or expected outcomes. These can be stated in a variety of ways and vary in 

nature with the subject area, level of instruction and school setting* In spite 

of this diversity, only a limited number of student behaviors related to thinking 

operations are believed to exist. By classifying the stated objective^ in terras 

of the intended cognitive behavior of students which is implied, the iritended 

4 

cognitive emphasis of any course of instruction can be described. If the 
order of importance of the objectives is specified, a ptofile can be made which 
shows the balance of emphasis on thinking operations intended. 

In contrast to what is intended, a teacher’s tests Represent one index of 
the kinds of activities actually practiced in the course. The teacher uses the 
tests as one way of determining whether the students are achieving the goals he 
holds important. Tests are also used to assign grades showing the student’s de*» 
gree of success in the course. While there may be objectives that remain 
untested, it seems quite likely that tests do represent the instructor’s actual 
emphasis in the course* In a similar manner to objectives, test questions can 
also be classified according to the cognitive behavior required of students* 

By using the weights the instructor used in scoring the tests, a profile of cog- 
nitive emphasis actually practiced in the course can be determined* The degree 
to which the cognitive profile of practices matches the intended cognitive pro- 
file is a measure of congruence of intent and practice in instruction* 

classification system, the Taxonomy of Intellectual Abilities (TIA) , based 
on Bloom’s Taxonomy , will be described in a later section of the paper* 
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It should be noted that this measure is just one indicator of the instruc*- 
tor’s emphasis. No statement of objectives or set of tests will ever be a com- 
plete description of intents and practices. For this reason it is wise to seek 
another index of intents and practices. Based on the Taxonomy of Intellectual 
Abilities (TIA) used to classify objectives and tests, the Cognitive Activities 
Rating Scale (CARS) lists a number of class activities. Several activities for 
each level of cognitive behavior the student might perform are included. To 
obtain a second index of the teacher’s intents for the course, the teacher could 
be asked to note the degree to which he would emphasize the various listed ac- 
tivities in the specific course he is teaching. Matching this indication of the 
activities he hopes to emphasize in the course with the profile based on his 
objectives would provide a measure of consistency of intents. A close match 
would imply that the instructor is sure of the emphasis he wishes to give and 

4' * 

t # # 

that he holds clearly defined and stable expectations for the cburse. 

Using the same set of class activities, the teacher’s students could be 
asked to indicate which activities they see as most characteristic or uncharac- 
teristic of the coiirse in question. Their responses as a group could be used to 
derive a second profile of the cognitive emphasis actually practiced in the 
course. The teach^t’s Intended emphasis on the Cognitive Activities Rating Scale 
could be matched with the Actual emphasis (as perceived by students) to provide 
a second index of Congruence of intent and practice in instruction. The two 

5 

samples of practice could be compared to obtain another measure of consistency. 
Figure 2 shows the data collected and compared in the CC Evaluation Procedure. 

I. — I 

^If the indices of practice are inconsistent, then it would be well to in- 
quire further into the instructor’s use of tests and attempt to obtain further 
information on how student learning is assessed. Not all tests are the paper and 
pencil variety. Sometimes the stuaeiit may be required to produce a particular 
product or perform a particular^ set of operations or demonstrate the acquisition 
of certain skills. These too are amenable to classification in terms of the im- 
plied cognitive behavior required of the students. 
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FIGURE 2 



STEPS IU70LVED IN THE CC EVALUATION PROCEDURE 



INTENTS 



PRACTICES 



Instructor’s Stated Objectives 






Course Test Questions 




CONGRUENCE 




(categorized by cognitive 
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(categorized by cognitive 


emphasis) 


emphasis) 



5 CONSISTENCY 
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6 CONSISTENCY 



Intended Cognitive Emphasis 
of Instructor 

(based on the Cognitive 
Activities Rating Scale) 



CONGRUENCE 
< 2 . 



Actual Cognitive Emphasis 
Perceived by Students 

(based on the Cognitive 
Activities Rating Scale) 



1-2 

3-4 

5-6 



Direct comparisons to assess congruence of intent and practice 
Indirect comparisons to assess congruence of intent and practice 
Comparisons reflecting consistency between the two measures of 
intent or practice 



Taxonomy of Intellectual Abilities 

Classification of objectives and tests in terms of the intended cognitive 
behavior of students vastly simplifies the analysis of instructional programs. 

It also enables direct comparisons across many grade levels, subject areas, 
and diverse school settings. 

The Taxonomy of Intellectual Abilities, shown in Figure 3, is the classifica- 
tion system used. The Taxonomy of Intellectual Abilities is based on Bloom’s 
Taxonomy of Educational Objectives: Cognitive Domain . The latter was first pub- 

lished in 1956 and a number of research studies have shown it to be a useful 
and effective tool. 

The Taxonomy of Intellectual Abilities, like Bloom’s Taxonomy, is hierar- 
chical in structure. This means that mental operations are ordered according 
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FIGURE 3 

TAXONOMY OP INTELLECTUAL ABILITIES 
(Adapted from Bloom's Taxonomy by Joe M. Steele) 

I. MEMORY 

Recall, recognition, bringing to mind of any kind of information. Some alter- 
ation of the material may be required, but this is a minor part of the task. 

Memory involves the ability to reproduce or recognize information as it was 
pr6S6n’fc6Q « 



IX 






Changing information into a different symbolic form to express the same idea, 

paraphrasing, pictures, graphs, summaries, outlines, or 
statements in technical or layman's language, it also Includes the use of 
metaphor, symbolism, and other non-literal statements. Translation involves 
ihe ability to comprehend information, including recasting or altering it in 



III . interpretation/extra polat ion 

Discovering and exploring the interrelationships among ideas (on a common-sense 
Comparing, contrasting, and explaining information based on the new 
view the perceived relationships provide. The task may require going beyond the 
given data in making inferences, predicting trends, and determining implications 
and consequences. Interpretation Involves the ability to extend and manipulate 
information to clarify relationships suggested by the data or to prolect trends 
based on patterns apparent in the data. 



IV. APPLICATION 

Utilizing abstractions (generalizations, rules, skills) in concrete situations, 
beiectlng and applying rules or methods to solve a specific problem, usually 
With a mlnimuin of direction or prompting as to which abstractions apply or how 
to use them. This kind of task gives practice in the Independent use of knowl- 
edge and skills, requiring the Identification of the issue as well as selection 
and use of the correct abstractions to solve problems in practical settings. 
Application Involves the ability to select the methods or generalizations called 

for by specific problem situations and perform the operations required to solve 
the problem. ' 



V. SYNTHESIS 

Recombining parts of previous experience with new material into a new Integrated 
whole, pattern, or structure not clearly there before. Synthesis implies a new 
product requiring original, creative thinking. This can take the form of a 
unique communication Involving skill in writing or speakingj a proposed set of 
operations, such as ways of testing hypotheses, or deve oping an effective plan 
to solve a complex^ problem^ or the derivation of abstract relations, as in 
making generalizations or mathematical discoveries. Synthesis Involves the 
ability to generate new ideas and solutions: inventing, designing, composing, 

cr69,’fclng 9 

VI. EVALUATION 

Clarifying and using a standard of appraisal in making judgments about the value 
of materials or methods for given purposes, in making judgments of good or bad, 
right or wrong, the standards or criteria used should be made explicit. This 
category forms a major link with the affective domain“l^ere values, liking, and 
enjoying are central processes. Evaluation is always somewhat subjective 
because either the standard cannot be proven to be correct or the idea to be 
judged cannot be proven to violate or Illustrate the standard. Evaluation in- 
volves the ability to develop and apply a set of standards for I’udging worth, 
and to support the judgments with a justification or rationale based on the 
criteria used. 

VII. FORMAL ANALYSIS 

Conducting a searching inquiry into the true nature of interrelationships and 
testing the validity of arguments according to appropriate ruD.es of reasoning, 
with conscious knowledge of the intellectual processes being performed. 

Analysis includes the ability to recognize unstated assumptions, distinguish 
facts from hypotheses and normative statements, recognize conclusions and pur- 
poses of the material, and check consistency, and relationships. Formal Analysis 
involves the ability to consciously apply appropriate rules of reasoning in 
testing the validity of statements, arguments, and conclusions. 
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to difficulty; each higher level of thinking requiring and including the lower 
thinking operations. The Taxonomy of Intellectual Abilities is composed of 
seven categories. The first three (Memory, Translation, Interpretation) are 
lower cognitive processes which lie in the general category Bloom calls Com- 
prehension. They represent the lowest level of understanding. Levels four 
through seven on the Taxonomy of Intellectual Abilities (Application, Synthesis, 
Evaluation, Formal Analysis) are higher cognitive processes. They are concerned 
with the use or development of intellectual skills and abilities. 

Bloom’s Taxonomy has primarily been used as a research instrument by 
trained observers. The intent of the Taxonomy of Intellectual Abilities adap- 
tation was to develop a classification system for use by untrained teachers 
and administrators. The description of each of the categories was repeatedly 
field tested and revised to lessen confusion between categories and produce 
appropriate interpretations of each thinking operation. 

A set of directions was developed which establishad conventions for dealing 
with ambiguous items or those which seemed to fit in more than one category. In 
addition, a sample set of objectives and questions was developed. This set of 
items serves both to illustrate the use of the Taxonomy of Intellectual Abilities 
and as a practice set for rudimentary training in the use of the system. (See 

Appendix. ) 

A reliability and validity study was conducted which demonstrated that the 
Taxonomy of Intellectual Abilities could be used reliably and with precision by 
untrained teachers and administrators.^ It was also shown that items were 

^A majority of the raters agreed on the classification of three-fourths of 
the practice items and about two-thirds of the real objectives and test items 
classified. A clear-cut assignment to a specific category was determined for 
79% to 89% of the items. When compared to the agreement among teachers in the 
grading of papers and similar tasks of this difficult nature, the degrde of agree- 
ment is seen to be relatively high. This result compares favorably with the 
study conducted by Julian Stanley (1957) using raters who received four weeks 
training in the use of Bloom’ s Taxonomy^ . 
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consistently associated with particular categories regardless of the rating 
characteristics of particular judges. 

Figure 4 illustrates the profiles of cognitive emphasis which result when 
an instructor’s objectives and tests are classified in terms of the Taxonomy 
of Intellectual Abilities. Instructor A intended to emphasize each cognitive 
level to soma degree, but sought to place about three-fourtb s of the emphasis 
on higher level thought processes. In his tests, 87% of the emphasis was placed 
on these levels and he did succeed in emphasizing six of the seven levels to 
some degree. 

Instructor B intended to emphasize four of the seven levels with slightly 
more than half of the emphasis placed on lower level thought processes. His 
tests emphasized only the lower cognitive levels with the lowest level 
(Memory) receiving over three-fourths of the emphasis. 

The Cognitive Activities Rating Scale 

The Cognitive Activities Rating Scale was developed to provide a more 
direct measure of the emphasis given to various cognitive activities than is 
provided by classifying objectives and test questions. This thirteen item 
forced-choice instrument is composed of short statements describing cognitive 
activities. Responses are made in terms of Strongly Agree, Agree, Disagree, or 
Strongly Disagree. Each response is made in terms of the degree to which the 
item is characteristic of the specific course of instruction involved. (See 
Figure 5 . ) 

The Cognitive Activities Rating Scale provides three profiles for each 
class, as well as scores useful for comparing classes. When administered to 
all students in a class, the mean and distribution of response indicates the 
instructor's actual practices insofar as they are perceived by the students as 
a group. When administered to instructors, each is asked to mark the responses 
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FIGURE 4 



PROFILES OF COGNITIVE EMPHASIS IN OBJECTIVES u TESTS 
(Actual Patterns for Two College Instructors) 



INSTRUCTOR A 



Levels of Thinking 



Percentage of Total Emphasis 
On Each Level in Objectives (0) and Tests (T) 

0 1 0 2 0 3 0 4 0 5 0 60 7 0 8 0 9 0 100% 




INSTRUCTOR B 



0 



10 20 30 40 50 60 70 80 90 100% 
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FIGURE 5 



COGNITIVE ACTIVITIES RATING SCALE (CARS) 



The following statements concern characteristics 
your course may have exhibited. For each statt?,- 
ment circle the letters which show the extent 
to which you agree or disagree. 

Base your answer on how well each sentence 
describes what is stressed in your class. 



RESPONSE 


CODE: 




Circle 


SA 


If YOU Stronglv Agree 
with the item 


Circle 


A 


If you Agree moderately 
with the item 


Circle 


D 


If you Disagree moderately 
with the item 


Circle 


SD 


If you Strongly Disagree 
with the item 



1. The course chiefly emphasizes knowledge of facts, 

2. Evaluation of issues and ideas is a central 
purpose of the course. 

3. Methods are tried out in problem situations. 

4. Going beyond the information given to see 
implications is a central concern. 

5. Emphasis is on logical reasoning and analysis. 

6. Restating ideas in your own words is stressed. 

7. Great emphasis is placed on memorizing. 

8. Problem-solving requiring original, creative 
thinking is stressed. 

9. The purpose of the '’ourse is to develop skill 
in using methods and ideas. 

10. Emphasis is placed on making inferences, 
deductions and interpretations. 

11. Critical analysis of studies is stressed. 

12 . 



13. 



A major purpose is to develop a standard for 
making judgments of value. 

Emphasis is on synthesizing information to 
develop new solutions and insights. 



CARS 9-11 

Copyright Joe M. Steele 1969 
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he would ideally like his class to give him. This measure is interpreted as an 

index of the instructor’s intended pattern of cognitive emphasis, . Another score 

is also obtained from the instructor. He is asked to indicate, the responses 

which he expects his class as a group will actually make. This is called the 

instructor’s Predicted Response and is an indicator of the instructor’s rapport 

and awareness of the way his presentation is being ’’received” by students. The 

Predicted Response also provides an unobtrusive measure of the emphasis the 

7 

instructor feels he is actually giving to various activities. 

The activities in the Cognitive Activities Rating Scale are keyed to the 
seven levels of the Taxonomy of Intellectual Abilities. In developing the scales 
a number of descriptions were generated and field tested by asking judges to 
classify them using the taxonomy. The final set of statements could be consis- 
tently classified at the appropriate cognitive level by persons with differing 
educational backgrounds. 

For six of the seven taxonomic levels, statements which express roughly 
the same concept were paired.^ By matching responses of the saBie individual 
to pairs of statements, the consistency of response can be ascertained. In 
scoring, two- thirds of the students in a class must show a consistent response 
to a pair for the level associated with that pair to be scored. The convention 
was adopted that for paired items with consistent responses, a clear-cut dire c - 
tion of response required fifty percent or more of the class to agree or disagree 

with both statements. Finally, the degree of emphasis is based on the mean 
response to the pair of statements. 

7one would expect the Predicted Response to fall somewhere between the Intended 
and Actual emphasis. The instructor should know the weaknesses in his presentation 
and the areas not clearly understood or misperceived by students. If his predic- 
tion lies farther from the students’ response than his Intents, this suggests that 
his awareness and perception of the situation is faulty. Feedback of such informa- 
tion to the teacher would allow a healthy correction in his view of his students. 

^For level II only one statement was retained in the final form of the instru- 
ment. At the time of administration field tests had failed to identify a second 
item which was associated more closely with level II than other levels. 
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The reason for these scoring rules is this: the lack of consistency of 
response would tend to indicate that those activities were either not clearly 
in evidence or that students were not certain of their opinions. Given consis- 
tent responses, the direction of response is a function of the proportion of 
the group involved and the intensity of their response, and both should be large 
enough to be judged representative of the group as a whole. 

Figure 6 illustrates the profiles of intended and practiced cognitive 

emphasis provided by the Cognitive Activities Rating Scale. This instrument in 

9 

itself provides much information meaningful to the teacher. Instructor A is 
seen by students as emphasizing six of the seven cognitive levels, with much 
emphasis given to five of them. All of the higher levels are emphasized. For 
five of the levels the instructor’s intended emphasis is realized even in res- 
pect to the degree of emphasis intended. One lower level process was strongly 
emphasized although the instructor wanted it to receive no emphasis. 

Students in Instructor B’s class perceived an emphasis on only one cogni- 
tive level, the highest level. However, they did strongly agree that activities 
representing three other levels were not characteristic of their class, although 
the instructor had intended to emphasize those three levels. He was seen as 
emphasizing only one of the six levels he intended to stress. 

The Cognitive Activities Rating Scale has been found to clearly discriminate 
among classes in terms of students’ perception of cognitive emphasis of presen- 
tation. Obtained reliability coefficients are at acceptable levels. Analysis 
of the cognitive factors show them to be receiving consistent responses to 

9 

For an expanded version of this instrument which includes noncognitive 
dimensions, see Joe M. Steele, Dimensions of the Class Activities Questionnaire , 
Illinois Gifted Program Evaluation, Center for Instructional Research and Curri- 
culum Evaluation, University of Illinois, Urbana, Illinois, Oct. 1969. 
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FIGURE 6 

PROFILES OF EMPHASIS ON THE COGNITIVE ACTIVITIES RATING SCALE 
(Actual Patterns for Two College Instructors) 



INSTRUCTOR A 



Levels of Thinking 



lOWER 

THOUGHT 

PROCESSES 



Memory 1 
Translation 2 
Interpretation 3 



Degree of Intended (I) and Actual (A) Emphasis 
Inconclusive None Some Much 
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HIGHER 

THOUGHT 

PROCESSES 



Application 4 
Synthesis 5 
Evaluation 6 
Formal Analysis 7 




INSTRUCTOR B 



Levels of Thinking 



LOWER 

THOUGHT 

PROCESSES 



Memory 1 
Translation 2 
Interpretation 3 



Degree of Intended (I) and Actual (A) Emphasis 
Inconclusive None Some Much 



HIGHER 

THOUGHT 

PROCESSES 







Application 4 
Synthesis 5 
Evaluation 6 
Formal Analysis 7 
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paired items and to be interpreted in the expected i?ay. Three of the seven 
levels (4, 6, and 7) are somewhat less clearly defined than the other four, 
but factors for all cognitive levels are judged to supply meaningful information. 



Congruence/Consistencv Matrices 

A method of summarizing the comparisons of intents and practices is used 
in the CC Evaluation Procedure which compares four assessments of congruence and 
two assessments of consistency per instructor or classroom group. Figure 7 
illustrates the relationships represented in this Congruence/Consistency 
Matrix. Judgments of congruence or incongruence are based primarily on a 
dichotomy of the Taxonomy of Intellectual Abilities into higher and lower 
cognitive levels. Decision rules have been carefully defined for making these 
judgments. This method of comparison is more lenient than one demanding a 

match across all seven levels of the Taxonomy of Intellectual Abilities. 
Thus the behavior of an instructor who approximates his intents would be judged 
congruent . 

Figures 5 and 6 have shown the major congruency comparisons of the two 
indices of intent and practice for two actual cases. The cases were two sections 
of the same undergraduate course. These are represented by the diagonal circled 
in Figure 7. It should already be obvious that Instructor A demonstrates a 
close match between intent and practice. Many contradictions between intent 
and practice as well as inconsistency in what is intended are revealed in 
Instructor B’s class. What becomes apparent in the matrix, however, is that 
this teacher’s intents as expressed in the Cognitive Activities Rating Scale 
do not match any of the other measures. Further than this, the instructor’s 
test questions assess the cognitive intents expressed in neither his objectives 
nor the Cognitive Activities Rating Scale. Close examination of the degree of 
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FIGURE 7 



ILLUSTRATIONS OF RELATIONSHIPS IN A CONGRUENCE/CONSISTENCY MATRIX 
(Actual Patterns for Two College Instructors) 



Key: T = Instructor's tests 

R = Students’ perception of instructor’s practices 

0 = Instructor’s stated objectives 

1 = Instructor’s intended cognitive emphasis for the course 
T 

Measures of instructor’s practices 

j Measures of instructor’s intent 

S = Consistency -S = Inconsistent 

G “ ConGruence -G = InconGruent 




and practice. 



match between objectives (0) and the students’ perception (R) reveals that 
little of what the instructor intended is seen as practiced. (The comparison 
is shown as congruent in the matrix because no clear contradictions exist although 
some are suggested.) 

The instructional treatment in case B remains quite ambiguous. Further 
clarification and refinement of this instructor’s class seems in order. More 
than this, as both classes represented the same course, the level of develop- 
ment of the instructional program represented by the course is called into 
question. No evaluation in terms of student outcomes seems appropriate until 
this program is made more cohesive and implemented more fully. 

Discussion 

Thc>' application of the CC Procedure makes use of a resource readily avail- 
able but seldom utilized by teachers: the perceptions of students. It requires 

the teacher to commit himself to an explicit set of priorities against which 
to view the goals and practices of the course. The findings (and the process 
of description involved) can be readily used by the teacher in modifying prac- 
tices, expectations, and his own perception of the instructional process. The 
procedure used as a formative evaluation process involving continued feedback 
and reaction should result in an instructional program which is highly congruent 
— one in which practice approximates intent. As the teacher clarifies his goals 
and makes judgments concerning the value and emphasis to assign to them, it is 
hoped that other kinds of evaluation will also be occurring. These related 
asst. "(Sments should ask questions such as: 

”How worthwhile are these objectives?” 

”How well does the student achieve the goals of the course?" 

"What are other outcomes of the course on students and others?" 

"How does this course compare with similar ones?" 
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The CC Procedure informs of the existence and implementation of an instruc- 
tional treatment. It provides the individual instructor with feedback on his 
classroom behavior and the perceptions of his students. In addition, the pro- 
cedure may serVe the following functions; 

1. An administrator can study the overall pattern of emphasis that 
characterizes a multi-section course, a department, or school. 

2. The procedure provides a dimension which renders comparable classes 
of differing levels and subject areas. 

3. The application of the procedure and reporting of results can serve 
as a basis for faculty dialogue, in-service training, or program 
development. Teachers can a) become exposed to the range of 
instructional treatments in use; b) deal with some of the standards 
and value judgments which underlie instruction; c) attempt to modify 
the patterns of emphasis bein_ practiced. 

There are also applications of the procedure which seem highly inappro- 
priate i. This procedure should not be used to evaluate teachers. When con- 
sidered in terms of teacher effectiveness, inconsistency and incongruence are 
not xiecessarily evidence of poor instruction. Contradictions which are exposed 
may reflect attempts to adjust practices or revise expectations. There are 
many dimensions of teaching not contained in the CC procedure which should 
enter into a judgment of the teacher. In addition, teachers should not be 
required to conform to a certain pattern of emphasis. Studies regarding the 
effectiveness of various patterns of emphasis remain to be conducted. 

Although its use to judge teachers is inappropriate, the procedure does 
suggest implications for effective teaching. A number of studies have revealed 
an imbalance in cognitive emphasis in which higher level processes are slighted 
and the lowest levels receive heavy emphasis. (Wood^®, Scannell and 



lOwood, Jean Marie. A Survey of Objectives for Teacher Education . Pre- 
pared for the Commission on Teacher Education, Association for Supervision and 
Curriculum Development. San Bernadino, California: San Bernadino City School 

System, 1960. 
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Stellwagen-^-^ , Lawrence-^"^ , Gallagher ) Some studies suggest that emphasizing 

14 

higher thought processes results in higher student achievement. (Hunkins ) 

A commoi^ fallacy of teachers af all levels from elementary to college is the 
belief that lower cognitive abilities should be emphasized in introductory 

1C 1 ^ 

classes and classes of lower ability. (Torrance , Pfeiffer °) 

17 

Deshpande and Webb in their studies of the relationship between teaching 
effectiveness and various aspects of instructor teaching goals, find that 
ineffective instructors give considerably greater importance to the goal of 
teaching facts than do effective instructors, who give highest importance to 
higher cognitive skills. (While it is probably not desirable to always empha- 
size the highest cognitive level, what does seem appropriate is that a balance 
of emphasis is achieved over many levels.) Another finding in the Deshpande 



^^cannell. Dale P. and Walter R. Stellwagen. Teaching and Testing for 
Degrees of Understanding. California Journal of Instructional Improvement , 
1, 1, 1960. 



^^Lawrence, Gordon D. Analysis of Teacher-made Tests in Social Studies 
According to the Taxonomy of Educational Objectives. Unpublished paper in 
the Clarmontiana Collection. Claremont, California: Honnold Library of the 

Claremont Colleges, 1963. 

^^Gallagher, James J., Mary Jane Aschner, and William Jenne* Productive 
Thinking of Gifted Children in Classroom Interaction . Washington, D. C.s 
Council for Exceptional Children, 1967. 

^'‘^Hunkins, Francis P. The Influence of Analysis and Evaluation Questions 
on Critical Thinking and Achievemeht in Sixth Grade Social Studies . Unpub- 
lished doctoral dissertation. Kent, Ohio; Kent State University, 1966. 

^^Torrance, E. Paul. Mental Health and Constructive Behavior . Belmont, 
California: Wadsworth Publishing Co., 1965. 

16pfeiffer, Isobel and 0. L. Davis, Jr. Teacher-Made Examinations: What 

Kind of Thinking Do They Demand, NASSP Bulletin , Sept. 1965. 

l^Deshpande, Anant S. and Sam C. Webb. Student Perceptions of Instructor 
Teaching Goals . Research Memoranda 67-5, 67-9, 68-5, and 68-6, Office of 
Evaluation Studies, Georgia Institute of Technology, Atlanta, Georgia, 1967 and 
1968. 
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and Webb studies is that there is greater teacher-student agreement on the 
emphasis given to goals for effective instructors. 

The CC Procedure could be utilized in research to further explore the 
relationship between patterns of emphasis and student achievement or congruence 
and effective teaching. 



APPENDIX 



Classification of Objectives and Questions in 
Terms of the Taxonomy of Intellectual Abilities 



Instructions for Judges 

Assign each item to one of the seven categories shown on the Taxonomy. 
You are to consider each item in terms of the ability the student uses to 

perform the task. Ask yourself, "What is the student asked to do?” If 

you are in doubt and feel that an item could be placed in more than one 
category, use the category with the higher Roraapi numeral. There is no set 
number of items per category and some categories may remain empty; however, 
each item must be placed in a category. Follow the steps outlined below: 

1. Read the category descriptions carefully — avoid applying your own 
definitions to the names of the categories. 

2. Read the notes on the form of goal statements and the classification 
of test questions for special decision rules. 

3. Now classify the Practice Objectives, items 1 - 20, considering each 

item in terms of the response the student must make. 

4. After assigning each item in the Practice Objectives to a category, 
check your accuracy and understanding by looking at the correct category 
numbers shown after item 20. 

5. Attempt to determine the rationale for assignment of any items you 
may have misassigned. It is important that you apply the same interpreta- 
tion as that indicated by the answer key. 

6. Next, classify the Practice Questions, items 21 - 40. Consider each 
question in the context of a college survey course >a exceptional children. 
Ask yourself what cognitive ability the student must use to perform the 
task. In cases of doubt about what preparation there, might have been 

for a question, classify the item in the highest category that seems to 
apply. 

7. Again check your understanding and accuracy by looking at the correct 
category assignments shown after item 40. 

8. Review the seven category descriptions before you begin classifying 
actual objectives and questions. 
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Notes on the Form of Goal Statements 



Goals are frequently stated in the form of predicates or phrases lacking 
subjects and sometimes lacking verbs. In many cases, however, a referent 
is implied by the structure of the statement. 

For example, objectives beginning "Familiarity with," "Be aware of," and 
"Know some of" seem to have the student as a refer^tnt. The antecedent "The 
student should (have)" seems to be implied. In such cases the verb will often 
indicate a cognitive activity. 

A second form of goal statement seems to refer to what the instructor 
must do in a course. Examples of this form might begin with "Instill a desire" 
and "Encouragement of." Statements of this tyt>e arfe often concerned with the 
student’s attitudes, feelings, and value systems. If they seem to call for 
the student to arrive at a decision or judgment after considering the facts, 
then the convention has been adopted to classify the statement in Level VI, 
Evaluation. Note that the major emphasis of an objective may be in the Affective 
rather than the Cognitive domain. As the procedure calls for all objectives 
to be assigned to a cognitive category, the level judged most apt is Evaluation. 

Sometimes the goal statement will call for the student to "try out" 
activities or "develop skill (in Using)" methods or ideas. Here the objective 
refers to actual performance and motor skills in the Behavioral domain. The 
convention followed is to classify behavioral objectives in Level IV, Application. 

Occasionally an objective will call for the student to simply be exposed 
to an experience. Such statements often begin "Exposure to," "Experience (with)," 
"Observe." Such Experiential objectives are also classified in Level IV, 
Application, 

Finally, goal statements frequently seem to imply the antecedent "Course 
content (includes)...." Examples of statements of this kind are "Characteristics 
of exceptional children" and "Conflicts in Special Education." Objectives 
commencing "To provide" also suggest a listing of content dimensions — the 
facts, concepts, or activities to be included in a course — rather than 
how the student is expected to deal with this content. 

If nothing at all is implied concerning the operations the student is asked 
to do, the assumption is made that "the student should have knowledge of ..." 
and the appropriate classification in Level I, Memory. If any hint is given of 
the ability the student is called on to use, the statement is classified in 
the highest level that seems to apply. 

Often times an educator assumes that an objective has been stated when 
only a topic has been listed. For example, "Conflicts in Special Education" 
says nothing about what the student is expected to do. In the absence of 
clarification, this can only be categorized as an objective stated in terms of 
content with the apparent implication that "the student should have knowledge 
of "such conflicts." This would be classified as requiring the lowest cogni- 
tive level: Memory. Indeed, the instructor may not have intended any more 

than that "the teacher should present" conflicts within Special Education. 
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This weakens the objective to the point where the goal seems to be simply 
exposure. This topic could be developed into a number of objectives, such as; 

a. The student is able to list three conflicts in the field of Special 
Education. (Level I, Memory) 

b. The student is able to state in his own works opposing viewpoints 
regarding several conflicts in Special Education (Level II, Translation) 

c. The student contrasts information presented to discover conflicts in 
Special Education. (Level III, Interpretation) 

I 

d. The student can identify conflicts in Special Education and is able 
to develop a plan for resolving one of these conflicts. (Level V, 
Synthesis) 

e. The student can cite conflicts in Special Education and state his own 
position regarding these conflicts, clearly stating the criteria on 
whicn he makes his judgment. (Level VI, Evaluation) 

f. The student is able to analyze the arguments and opposing points of 
view regarding one of the conflicts in Special Education and explore 
assumptions and conclusions in terms of their logical consistency. 

(Level VII, Formal Analysis) 

In many cases the instructor will have made no conscious attempt to con- 
ceptualize what he is doing in terms of the implications for student outcomes. 

It can be taken as a general rule that a specific objective for a particular 
course will generally be stated in a form which reflects its refinement and 
degree of clarification. The instructor may require time and opportunity for 
discussion to develop his statement of objectives to a level which he finds 
satisfactory. 



Notes on Classifying Test Questions 

Classifying questions is a somewhat different operation than classifying 
objectives — for several reasons. First, some questions are not simply of 
one kind but part of several categories. As Norris Sanders^® points out, however, 
difficulty of classification is no detraction from the quality of the question. 

Second, the form of the question tends to lead to certain kinds of thinking. 
Thus, many true-false questions can only be memory questions calling for the 
student to recall whatever answer the teacher endorsed. 

Third, the classification of a question depends in part on the amount of 
information that has been presented by the teacher or is a part of the student’s 
background. The question may appear to call for evaluation by the student, but 
if the instructor has already indicated the judgment he prefers, the operation 
may be merely recall. 

The person who classifies the objectives and questions will usually be 
aware of their context. He will know if the course is introductory or advanced. 



^^Sanders, Norris M. Classroom Questions; What Kinds? New York: Harper 

& Row, 1966. 
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whether the test is the first or the final one and what has appeared on other 
tests, and probably will know something about the organization of the subject 
matter. Based on this context, questions must be classified in terms of the 
level of thinking they seem to suggest, apart from what the instructor intends 
or the nature of the student’s response. It should be noted that such an 
approach may provide an overestimate of the cognitive levels emphasized. This 
is consistent with the convention to always resolve a classification problem 
in favor of the higher category involved. 



Practice Objectives 

1. The course chiefly emphasizes knowledge of facts. 

2. Evaluation of issues and ideas is a central purpose of the course. 

3. Methods are tried out in problem situations. 

4. Tests are based primarily on lectures. 

5. The course emphasizes a detailed study of the underlying factors 
contributing to the educational problems of children. 

6. Going beyond the information given to see implications is a central 
concern. 

7. Emphasis is on logical reasoning and analysis. 

8. Students are expected to independently explore and initiate activities 
in the course. 

9. Restating ideas in your own words is stressed. 

10. Great emphasis is placed on memorizing. 

11. Problem-solving requiring original, creative thinking is stressed. 

12. Chief emphasis is on explanation and summarization of information. 

13. The purpose of the course is to develop skill in using methods and 
ideas. 

14. Students are expected to comprehend the content and be able to state 
it in various ways. 

15. The course utilizes life-like settings for identifying and s-olving 
problems faced by the child. 

16. Emphasis is placed on inferring, deducing, and in^;erpreting. 

17. Students are encouraged to discover and develop many possible 
solutions to problems. 
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18. 



Critical analysis of studies is stressed. 

19. A major purpose is to develop a standard for making judgments of 
value, and applying it. 

20. Emphasis is on synthesizing information to develop new solutions 
and insights. 



Practice 


Objectives 


Key: 


1. I 


5. 


VII 


9. 


2. VI 


6. 


III 


10. 


3. IV 


7. 


VII 


11. 


4. II 


8. 


V 


12. 



II 


13. 


IV ■ 


17. 


VI 


I 


14. 


II 


18. 


VII 


V 


15. 


IV 


19. 


VI 


III 


16 . 


III 


20. 


V 



Practice Questions 

21. Compare and contrast the following terms and ideas: a) Gifted “ Creative 

b) a handicap a 
disability 

22. Nominate the person (s) you feel have made a significant contribution 
to the fields of (a) the Gifted and (b) the Emotionally Disturbed in 
the past two decades. Substantiate your choices. 

23. Disorders of voice are characterized by three main types: 

a) pitch, hoarseness, duration 

b) quality, intensity, pitch 

c) monotone, nasality, pitch 

d) gutteral, quality, breathiness 



24. 



25. 



Convergent thinking might best be diagrammed as: 



a) 




b) 



t 



c) 



I 



d) 



i 



In evaluating screening techniques for the gifted, the term "effective- 
ness” is defined as the ratio of gifted correctly identified to the 
total number of actually gifted in the group. Restate this concept 
in your own words or illustrate it in a formula, diagram, or chart. 



26. T-F The field of Learning Disabilities is not really a field of special 
education. State your rationale for your answer. 

27. A retarded boy whose chronological age is 8 years-4 months, has a mental 
age of 5 years-10 months as measured by the Stanford-Binet Intelligence 
Scale. Using the formula IQ = MA/CA x 100, state the boy's intelligence 
quotient. 

28. An experimental group test to discover underachieving gifted children has 
been developed. The largest sample tested to date produced these results 

No. Tested No.Ident. as Underach. Gif ted Correctly Ident. Overlooked 
10,150 ’ 120 30 60 



The test's effectiveness is: a) 25% b) 60% c) 30% d) 33% e) 90% 



29. You have been given $100,000 to develop a test of creativity to be 
used in the schools. Explain in some detail how you would approach 
the problem, how you would benefit from the mistakes of others, and 
describe in general terms what you would expect your instruments to 
look like when you were finished. 

30. Define etiology. 

31. List all the criticisms you can that can be leveled at the following 
research: 

"Five physically handicapped children, see in LU’s research hospital 
out-patient clinic, were given one gram daily of a new type of 
’muscular control’ drug, for a period of 100 to 150 days. The 
results were excellent in every case: mothers reported the children 

were able to do more things and they ’seemed much brighter.’ This 
V7as confirmed by psychological testing: 

CA Range Ave. CA WISC IQ Before Drug WISC 10 After Drug 

3-2 to 9-9 5.0 87.12 99.99 



IQ Chan}2;e Sig. Level 

12.87 .01 

Conclusion: We therefore recommend that this drug be used extensively 

with physically handicapped children.” 

32. T-F According to Quay, a personality disorder among the emotionally 
maladjusted is characterized by aggressiveness, lying, and defiance of 
authority. 

33. What evidence can you cite from your reading and class discussion to 

support or refute the following generalization: "There is a direct 

relationship between social class and emotional disturbance.” 

34. How would you go about testing the following hypothesis: Hard-of-hearing 

children are as defective in auditory decoding as are partially seeing 
children in visual decoding? Start from scratch — you have no standard- 
ized tests for auditory or visual decoding processes. 

35. Discuss the following comparisons: The retarded are to normal as 

normals are to gifted. 

36. You will be shown a videotape of an exceptional child in school. Some 
information concerning teacher’s observations and test results are 
available upon request for specific information. Based on our study 
of exceptional children, make a tentative diagnosis, identifying the 
major educational problems of this child and suggesting possible steps 
to take in dealing with him. 

37. Suggest at least three specific and concrete examples of how to im- 
prove teacher training programs in the area of the gifted. Explain in 
each case why you feel this is an improvement on existing practice. 
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38. Indicate briefly the purpose, methodology, and results reported in 

Ringness’ study of emotional adjustment of academically successful and 
non-successf ul boys. Discuss the weaknesses and shortcomings of this 
study and suggest what improvements in design and procedure could be 
made in testing the problems posed by his study. 



39. We have studied the characteristics of exceptional children and pro- 
visions for their education. Now assume you have just been appointed 
as the first Director of Special Education in the typical American 
community of Anurba, population 30,000. Outline the requirements 
for special education classes in this community’s schools. 

40. Discuss the educationally relevant information in the following profiles: 



Grade Level 
Age Equivalent 
Life Age 
Height 
Weight 

Mental Ability 
Reading 
Arithmetic 
Spelling 

Self-sufficiency 
Social Adjustment 




Ann 



Practice Questions Key: 



21. 


Ill 


25. 


II 


29. 


22. 


VI 


26. 


VI 


30. 


23. 


I 


27. 


II 


31. 


24. 


II 


28. 


III 


32. 



V 


33. 


VI 


37. 


V 


I 


34. 


V 


38. 


VII 


VII 


35. 


III 


39. 


IV 


I 


36. 


IV 


40. 


IV 
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